A commutative but not cocommutative graded Hopf algebra $\Hn$, based on ordered rooted trees, is studied. This Hopf algebra generalizes the Hopf algebraic structure of unordered rooted trees $\Hc$, developed by Butcher in his study of Runge--Kutta methods and later rediscovered by Connes and Moscovici in the context of non-commutative geometry and by Kreimer where it is used to describe renormalization in quantum field theory. It is shown that $\Hn$ is naturally obtained from a universal object in a category of non-commutative derivations, and in particular, it forms a foundation for the study of numerical integrators based on non-commutative Lie group actions on a manifold. Recursive and non-recursive definitions of the coproduct and the antipode are derived. It is also shown that the dual of $\Hn$ is a Hopf algebra of Grossman and Larson. $\Hn$ contains two well-known Hopf algebras as special cases: The Hopf algebra $\Hc$ of Butcher--Connes--Kreimer is identified as a proper subalgebra of $\Hn$ using the image of a tree symmetrization operator. The Hopf algebra $\Hf$ of the Free Associative Algebra is obtained from $\Hn$ by a quotient construction.